Got a tip for us?
Let us know
Become a MacRumors Supporter for $50/year with no ads, ability to filter front page stories, and private forums.
Top 50 Posters Extrapolated, July 2019
- Thread starter chown33
- Start date
- Sort by reaction score
This post presents the intermediate post count statistics, measured somewhat after the mid-point between July 03 and Jan 03. It's briefer than the usual semi-annual stats post. It's later than the actual midpoint because I was busy with other things.
I plan to post the graphs and their associated remarks later.
All measurements taken 13 October 2019 by chown33.
Table 1: Current Top 50 Posters
Table Columns:
Remarks
Table 2: Currently Active Top 50 Posters
Table 3: Top 50 Prolific Posters
Table Columns:
I plan to post the graphs and their associated remarks later.
All measurements taken 13 October 2019 by chown33.
Table 1: Current Top 50 Posters
Code:
Current Top 50 Cur 3-Mo Cur Delta Prev Delta 3-Mo Foll- Avg of one
Rank on 2019-10-13 Posts Posts Rate Rate Rank Rank Likes owers post every
---- ---------------- ----- ----- ----- ------ ---- ----- ----- ----- ----------
1 maflynn 63830 577 5.66 -4.23 1 0 1018 25 4h 14.6m
2 C DM 47420 591 5.79 -0.92 2 0 398 22 4h 8.5m
3 Scepticalscribe 45292 3003 29.44 +2.82 4 +1 1526 40 0h 48.9m
4 GGJstudios 44360 9 0.09 -0.69 3 -1 38 15 11d 8h 0.0m
5 Applejuiced 40650 --- --- --- 5 0 1 143 ---
6 Doctor Q 36029 225 2.21 -1.21 6 0 268 42 10h 52.8m
7 GoCubsGo 35743 2 0.02 +0.02 7 0 4 8 51d 0h 0.0m
8 Apple fanboy 32686 1850 18.14 -1.66 9 +1 1122 15 1h 19.4m
9 mad jew 32194 --- --- --- 8 -1 1 4 ---
10 Relentless Power 30289 1389 13.62 -1.82 12 +2 1349 20 1h 45.7m
11 mkrishnan 29641 --- --- --- 10 -1 1 11 ---
12 Weaselboy 29118 565 5.54 +1.04 13 +1 513 81 4h 20.0m
13 Eidorian 29081 --- --- -0.03 11 -2 0 63 ---
14 rdowns 27345 2 0.02 +0.02 14 0 2 24 51d 0h 0.0m
15 SandboxGeneral 24931 478 4.69 +3.16 16 +1 765 23 5h 7.3m
16 robbieduncan 24480 --- --- --- 15 -1 3 23 ---
17 Abstract 24378 1 0.01 -0.02 17 0 2 7 102d 0h 0.0m
18 jsw 22817 --- --- -0.01 18 0 0 2 ---
19 Mr. Anderson 22407 --- --- --- 19 0 0 3 ---
20 Hellhammer 22076 --- --- --- 21 +1 0 53 ---
21 alphaod 22046 28 0.27 -0.55 22 +1 61 5 3d 15h 25.7m
22 Blue Velvet 21652 --- --- -0.01 23 +1 13 2 ---
23 MacNut 21537 197 1.93 -0.54 24 +1 182 7 12h 25.6m
24 eyoungren 21156 321 3.15 -0.60 25 +1 496 120 7h 37.6m
25 Rogifan 20643 558 5.47 +1.48 26 +1 1948 18 4h 23.2m
26 I7guy 20373 1099 10.77 -3.77 32 +6 758 4 2h 13.6m
27 Jaffa Cake 19801 --- --- --- 27 0 1 16 ---
28 bartelby 19794 --- --- --- 28 0 0 23 ---
29 MacDawg 19708 110 1.08 +0.02 29 0 244 3 22h 15.3m
30 iGary 19583 --- --- --- 30 0 0 6 ---
31 Peace 19464 26 0.25 -0.98 31 0 34 5 3d 22h 9.2m
32 The Game 161 19373 883 8.66 +0.31 40 +8 669 10 2h 46.3m
33 miles01110 19264 --- --- --- 33 0 0 1 ---
34 PlaceofDis 19232 --- --- --- 34 0 0 3 ---
35 iBlue 19174 --- --- --- 35 0 0 11 ---
36 edesignuk 19077 --- --- --- 36 0 0 1 ---
37 balamw 19075 --- --- -0.01 37 0 0 12 ---
38 Intell 18872 4 0.04 -0.05 38 0 5 24 25d 12h 0.0m
39 Nermal 18676 49 0.48 -0.24 39 0 43 13 2d 1h 57.6m
40 ~Shard~ 18388 --- --- -0.01 41 +1 0 1 ---
41 AidenShaw 18069 110 1.08 -1.67 43 +2 89 13 22h 15.3m
42 dukebound85 18043 25 0.25 +0.07 42 0 74 --- 4d 1h 55.2m
43 IJ Reilly 17915 --- --- --- 44 +1 14 2 ---
44 ucfgrad93 17525 134 1.31 -0.76 46 +2 495 9 18h 16.1m
45 DoFoT9 17494 --- --- --- 45 0 0 122 ---
46 kdarling 17377 --- --- -0.36 47 +1 0 18 ---
47 Fishrrman 17152 773 7.58 -0.81 56 +9 423 15 3h 10.0m
48 Huntn 16995 550 5.39 +0.94 54 +6 1323 32 4h 27.1m
49 jav6454 16864 75 0.74 +0.67 49 0 67 9 1d 8h 38.4m
50 -aggie- 16793 1 0.01 -0.16 48 -2 1 19 102d 0h 0.0mTable Columns:
- Rank = Position in order by post count, 1 (highest) to 50 (lowest).
- Current Top 50 = members with the 50 currently highest post counts.
- Cur Posts = current count of posts by this member (see Note below).
- 3-Mo Posts = count of posts made in this measurement period, or "---" if none.
- Cur Rate = average # posts per day by this member over the measurement period.
- Delta Rate = change in average daily post rate from the prior semi-annual period to the latest period.
- Prev Rank = previous rank, as of 3 months ago (July 2019); a number above 50 for those who are new to the Top 50.
- Delta Rank = change in rank, i.e., # of slots +higher or -lower in rank since prior semi-annual stats.
- 3-Mo Likes = count of likes received in this measurement period.
- Followers = current count of forum followers, or "---" if data unavailable.
- Avg of one post every = average time between member's posts, in days, hours, and minutes, based on a 24-hour day.
the Politics, Religion, Social Issues forum
the Console Games forum
the Code Sharing and Software Promos forum
the Marketplace forum
the Wasteland
the Console Games forum
the Code Sharing and Software Promos forum
the Marketplace forum
the Wasteland
Remarks
- As noted in previous updates since July (see posts in this thread), there were several changes in the Top 10. @Scepticalscribe moved past @GGJstudios, and both @Apple fanboy and @Relentless Power moved into the Top 10.
- The action just outside the Top 10 continues, with @Weaselboy and @SandboxGeneral both moving up.
- Members @Fishrrman and @Huntn moved into the Top 50, and @OllyW slipped to rank 51.
- Among users previously in the Top 50, @The Game 161 moved up 8 positions, the largest change in rank.
Table 2: Currently Active Top 50 Posters
Code:
Active Top 50 Cur 3-Mo
Rank on 2019-10-13 Posts Posts
---- ---------------- ----- -----
1 maflynn 63830 577
2 C DM 47420 591
3 Scepticalscribe 45292 3003
4 GGJstudios 44360 9
5 Doctor Q 36029 225
6 GoCubsGo 35743 2
7 Apple fanboy 32686 1850
8 Relentless Power 30289 1389
9 Weaselboy 29118 565
10 rdowns 27345 2
11 SandboxGeneral 24931 478
12 Abstract 24378 1
13 alphaod 22046 28
14 MacNut 21537 197
15 eyoungren 21156 321
16 Rogifan 20643 558
17 I7guy 20373 1099
18 MacDawg 19708 110
19 Peace 19464 26
20 The Game 161 19373 883
21 Intell 18872 4
22 Nermal 18676 49
23 AidenShaw 18069 110
24 dukebound85 18043 25
25 ucfgrad93 17525 134
26 Fishrrman 17152 773
27 Huntn 16995 550
28 jav6454 16864 75
29 -aggie- 16793 1
30 OllyW 16747 62
31 WildCowboy 16717 95
32 samcraig 16609 37
33 gnasher729 16497 73
34 cube 16430 504
35 Sun Baked 14874 5
36 KPOM 14378 189
37 arn 14076 161
38 cmaier 14048 691
39 Tech198 13944 289
40 Mitthrawnuruodo 13584 84
41 Lord Blackadder 13518 109
42 Mlrollin91 13475 129
43 mattopotamus 13219 329
44 annk 12882 117
45 lordofthereef 12867 36
46 h9826790 12766 374
47 jamezr 12517 306
48 CanadaRAM 12419 1
49 aristobrat 12244 18
50 apolloa 12221 513Table 3: Top 50 Prolific Posters
Code:
Prolific Top 50 Cur 3-Mo
Rank on 2019-10-13 Posts Posts
---- ---------------- ----- -----
1 Scepticalscribe 45292 3003
2 Apple fanboy 32686 1850
3 Relentless Power 30289 1389
4 EugW 7292 1218 *
5 I7guy 20373 1099
6 AustinIllini 10724 1080 *
7 Shirasaki 9409 939 *
8 The Game 161 19373 883
9 Zenithal 8989 870 *
10 Fishrrman 17152 773
11 cmaier 14048 691
12 C DM 47420 591
13 maflynn 63830 577
14 Weaselboy 29118 565
15 GutWrench 3885 563 *
16 Rogifan 20643 558
17 Huntn 16995 550
18 newellj 7230 530 *
19 Clix Pix 8582 517 *
20 apolloa 12221 513
21 deeddawg 8190 510 *
22 cube 16430 504
23 Abazigal 11582 496 *
24 Shanghaichica 11109 493 *
25 dannyyankou 8591 480 *
26 SandboxGeneral 24931 478
27 LizKat 5306 477 *
28 GrumpyMom 8308 462 *
29 akash.nu 7337 448 *
30 The-Real-Deal82 7679 442 *
31 BasicGreatGuy 11663 420 *
32 h9826790 12766 374
33 DeltaMac 9523 371 *
34 D.T. 8917 370 *
35 macduke 10566 359 *
36 justperry 10119 343 *
37 mi7chy 5926 339 *
38 kazmac 7750 338 *
39 chabig 5868 331 *
40 mattopotamus 13219 329
41 eyoungren 21156 321
42 chrfr 8172 316 *
43 rui no onna 6797 310 *
44 jamezr 12517 306
45 chown33 8345 301 *
46 Tech198 13944 289
47 redheeler 7393 269 *
48 willmtaylor 10185 267 *
49 deconstruct60 7973 267 *
50 Knowlege Bomb 6515 255 *Table Columns:
- Rank = Position in order by post count (Table 2) or post rate (Table 3), 1 (highest) to 50.
- Active Top 50 / Prolific Top 50 = members with the 50 currently highest post counts or post rates among those who have posted in the last 3 months.
- Cur Posts = count of posts by that member (uses the same criteria as Table 1 for which posts are counted).
- 3-Mo Posts = count of posts in this measurement period.
- * = this user is in the Prolific Top 50 list, but not in the Active Top 50 list.
The above update is the first statistics post since the site converted to XenForo 2.1. I had to make some changes in how the statistics were collected, and determine how the new "reaction scoring" system worked.
The new term used in a member's profile info is "Reaction score", instead of "Likes". I've kept the term "Likes" in the above tables, for the reasons below.
The button that lets you Like a post now gives you a total of 6 different reactions, at least 2 of which have fairly plain negative connotations.
In the underlying XenForo system, one's "Reaction score" is the net sum of positive and negative reactions. However, MacRumors doesn't currently offer any negative reactions. The scoring is:
+1 : like love haha wow
+0 : sad angry
So although the sad or angry reactions appear to be negative, they simply don't count when calculating a user's total reaction score. Someone can have 100's of sad or angry reactions, and it won't change their Reaction score at all.
EDIT
Based on info here, and a quick test, I moved the wow to +1 score.
The new term used in a member's profile info is "Reaction score", instead of "Likes". I've kept the term "Likes" in the above tables, for the reasons below.
The button that lets you Like a post now gives you a total of 6 different reactions, at least 2 of which have fairly plain negative connotations.
In the underlying XenForo system, one's "Reaction score" is the net sum of positive and negative reactions. However, MacRumors doesn't currently offer any negative reactions. The scoring is:
+1 : like love haha wow
+0 : sad angry
So although the sad or angry reactions appear to be negative, they simply don't count when calculating a user's total reaction score. Someone can have 100's of sad or angry reactions, and it won't change their Reaction score at all.
EDIT
Based on info here, and a quick test, I moved the wow to +1 score.
Last edited:
Graphs for post-counts and post rates.
Based on measurements taken 13 October 2019 by chown33.
Info About All the Graphs
The horizontal (X) axes represent dates, and are labeled as YYYY-MM-DD.
The vertical (Y) axes are automatically determined, and are specific to each graph. The upper and lower limits of a graph are based on the range of post-counts or post-rates in the subset of graphed data. As a result, the steepness of line slopes can't be compared between different graphs.
Colors are also automatically assigned, from a rotating pool of 12 colors. This can make color-to-name conversion ambiguous when more than 12 members are graphed.
Use each graph's legend or key on the right to disambiguate duplicate colors. The legend is arranged in the same order as the right-most points of each graph. For example, if a graph has the upper red line as maflynn (top of legend), then the lower red line is someone else.
A dashed line is automatically assigned to inactive users (no posts in the latest measurement period), and a solid line to active users (at least 1 post in latest period). To reduce the clutter, the only inactive user typically shown is @mad jew.
Graphs 1: Active Top 50 Posters - 2 Groups, 1 Composite
![GRAPH-01--top-01-25.gif]( "GRAPH-01--top-01-25.gif")
![GRAPH-01--top-26-50.gif]( "GRAPH-01--top-26-50.gif")
![GRAPH-06--active-50.gif]( "GRAPH-06--active-50.gif")
These graphs represent Active Top 50 Posters, i.e. those with the highest post counts. @mad jew is the only inactive member shown, for historical reasons.
Member @I7guy appears in both Group graphs, with different line slopes, showing that slopes between graphs aren't comparable.
Graphs 2: Prolific Posters, Weekly Post Growth - 3 Groups, 1 Composite
![GRAPH-12--growth-1.gif]( "GRAPH-12--growth-1.gif")
![GRAPH-12--growth-2.gif]( "GRAPH-12--growth-2.gif")
![GRAPH-12--growth-3.gif]( "GRAPH-12--growth-3.gif")
![GRAPH-30--growth-36.gif]( "GRAPH-30--growth-36.gif")
These graphs represent Prolific Posters, i.e. those with the highest post rates, regardless of their post count.
The post count of each member shown is usually sampled on the same day each week. The growth from the 1st sampled date is then drawn on the graph. Thus, the first value graphed is the increase in post count for the week ending on the date shown.
The Group graphs are less cluttered and should be easier to follow than the Composite graphs. The Composites are useful mainly to see everyone's relative counts or rates on the same scale.
Remarks
Based on measurements taken 13 October 2019 by chown33.
Info About All the Graphs
The horizontal (X) axes represent dates, and are labeled as YYYY-MM-DD.
The vertical (Y) axes are automatically determined, and are specific to each graph. The upper and lower limits of a graph are based on the range of post-counts or post-rates in the subset of graphed data. As a result, the steepness of line slopes can't be compared between different graphs.
Colors are also automatically assigned, from a rotating pool of 12 colors. This can make color-to-name conversion ambiguous when more than 12 members are graphed.
Use each graph's legend or key on the right to disambiguate duplicate colors. The legend is arranged in the same order as the right-most points of each graph. For example, if a graph has the upper red line as maflynn (top of legend), then the lower red line is someone else.
A dashed line is automatically assigned to inactive users (no posts in the latest measurement period), and a solid line to active users (at least 1 post in latest period). To reduce the clutter, the only inactive user typically shown is @mad jew.
Graphs 1: Active Top 50 Posters - 2 Groups, 1 Composite
These graphs represent Active Top 50 Posters, i.e. those with the highest post counts. @mad jew is the only inactive member shown, for historical reasons.
Member @I7guy appears in both Group graphs, with different line slopes, showing that slopes between graphs aren't comparable.
Graphs 2: Prolific Posters, Weekly Post Growth - 3 Groups, 1 Composite
These graphs represent Prolific Posters, i.e. those with the highest post rates, regardless of their post count.
The post count of each member shown is usually sampled on the same day each week. The growth from the 1st sampled date is then drawn on the graph. Thus, the first value graphed is the increase in post count for the week ending on the date shown.
The Group graphs are less cluttered and should be easier to follow than the Composite graphs. The Composites are useful mainly to see everyone's relative counts or rates on the same scale.
Remarks
Code:
9 Weaselboy 29118 565
11 SandboxGeneral 24931 478Will his reign of terror ever end?
I applaud - and am impressed by - your incredible work rate in putting these together.
Thanks for sharing them with us; long before I ever even featured on these graphs and lists, I must say that I found them fascinating reading.
Attention @SandboxGeneral please pick up your escaped penguin at the main entrance.
Thank you for finding Tux so quickly! I was terribly worried and I appreciate receiving your alert over the MacRumors PA system directing my attention to the main entrance to get him. I'm so relieved!
A nice read, chown33. I was pleased to see that @CanadaRAM is still around, still giving useful technical advice to other users.
Welcome to the October 2019 edition of "Fun with Graphs".
First we have the "Race Into Top 10" graph, which now includes @Weaselboy and @Eidorian, who are outside the Top 10. User @mkrishnan sits at #10, so that's the benchmark to pass.
![GRAPH-66--fun-1a.gif]( "GRAPH-66--fun-1a.gif")
Next we look at the top Likes getters.
![GRAPH-66--fun-2.gif]( "GRAPH-66--fun-2.gif")
Finally we take a different look at some users whose post-rates took off in early September.
For this graph I had to select some users by hand. I started with the ever-prolific ScepticalScribe for the "top line", and myself as a far less prolific poster. Then I picked a few "rabbits" who took off in September: @AustinIllini, @deeddawg, @EugW, and @newellj.
This is a "Posts per Week" graph, so we see the count of posts made in each week, rather than the cumulative growth for each week. In this graph, a week with no posts will appear as a point at 0 for a date. A flat line between two dates means the same number of posts were made each week.
![GRAPH-66--fun-10b.gif]( "GRAPH-66--fun-10b.gif")
First we have the "Race Into Top 10" graph, which now includes @Weaselboy and @Eidorian, who are outside the Top 10. User @mkrishnan sits at #10, so that's the benchmark to pass.
Next we look at the top Likes getters.
Finally we take a different look at some users whose post-rates took off in early September.
For this graph I had to select some users by hand. I started with the ever-prolific ScepticalScribe for the "top line", and myself as a far less prolific poster. Then I picked a few "rabbits" who took off in September: @AustinIllini, @deeddawg, @EugW, and @newellj.
This is a "Posts per Week" graph, so we see the count of posts made in each week, rather than the cumulative growth for each week. In this graph, a week with no posts will appear as a point at 0 for a date. A flat line between two dates means the same number of posts were made each week.
Prompted by a question from @SandboxGeneral in another thread, I realize I haven't described the How or Why behind the statistics graphs. I will now remedy that.
I produce the statistics graphs (and my haunting avatars) in PDF, which I then convert to GIF using the 'sips' command.
The How
The input to the graph-maker stage is a series of lines containing the data for a single user on a single date. The data for each user is collected into a polyline, which means a series of line segments that make up a user's line graph. In addition, min and max values are calculated across all users and dates, and that data is used to automatically determine the extent of the final graph.
This data is then fused with some configurable (but fairly stable) drawing data, such as dimensions, colors, line-widths, font sizes, etc. and that's all then fed into the actual PDF-generator.
The whole production is scripted. It's a combination of shell scripts (bash), awk scripts, and commands like sort, head, tail, tr, sips, etc. The whole thing is automated, so all I do is tell the script to use new data, such as what I collected on 13 October, and it runs through it all, producing a couple dozen PDF files in about 10 seconds.
If I want to adjust a particular graph, I edit one text file that determines which users and which dates to select from all the collected data. Then I run the script to produce all the graphs again. Yes, it unnnecessarily repeats the production of 20 or so PDF files, but since it only takes 10 seconds, I don't much care.
Once the PDF graphs look the way I want, I then convert them to GIF files using the 'sips' command. This is also done in a script, because there are two distinct graph sizes, and 'sips' needs to be told what output size for each group.
Finally, I write up the thrilling descriptions, and put it all together in a post on MacRumors.
In general, unless I have a completely new graph, or significant changes to past ones, the actual production of the graphs takes only a minute or so, because it's almost completely automated. From selecting the Top 50, sorting by Likes, splitting into groups, or setting graph titles, it's all in different scripts, and all I need to do is run it. The most time-consuming part is looking at the graphs, picking out interesting things to remark on, and writing the text of the posts.
The Why
When I first looked into producing graphs, I considered a few ways to do it.
One of the criteria was that it be automatable, meaning I could take a bunch of data output from an awk program (or other script) and turn it into a line graph. I was mainly interested in line graphs, because they're a natural fit when the data is counts obtained on dates.
One approach would be to use a graphing or charting package, such as the ones listed here:
en.wikipedia.org
I looked at a few, and downloaded a couple that offered compilable source, and built them. I also went through some user's guides, to see how to tell them what to do.
If I chose a charting package, even an automatable one, I'd have a lot of flexibility in picking different kinds of charts and graphs. Sadly, that flexibility wouldn't really be useful, because the most common graph I expected would pretty much always be a line graph.
I also considered what tools or knowledge I'd gain if I used an existing automated charting package. Basically, I'd know how to automate one charting package to produce line graphs. As useful as that would be for the purpose of producing graphs, that wasn't my only goal. To me, it seemed like a fairly narrow skill.
On the other hand, suppose I made a simple ad hoc graphing package of my own, which output to one of the image file formats:
en.wikipedia.org
It wouldn't need to be complex, with many different graph or chart types, because I'd be almost exclusively making line graphs.
I already knew that 'sips' can convert between several formats, so I could use a direct output format that was convenient for production, even if I uploaded a different format like GIF or PNG.
From what I already knew about PDF, it has a compact and easily represented way to draw polylines, set colors, and other simple drawing operations. It also has a lot of control over rendering text, which I knew I'd need for labels and such.
So if I wrote my own graph-maker that produced PDF, I'd learn a broader set of skills, namely, how to build simple line graphs from raw numbers, and how to draw things with PDF.
The What
The first stage was to turn a bunch of numbers, dates, and names into a halfway decent line graph. For that, I wrote scripts that produced output in a very simplified "intermediate language" that said things like:
I was fairly confident that once I got the basic graph production working, this intermediate form could be replaced by a real PDF generator.
The second stage was to learn enough basic PDF to draw a few things on a page. That turned out to need a bit of "housekeeping" in order to make a well-formed PDF file.
It turns out that a PDF file has some required elements, such as an xref table and multiple objects with streams that contain the actual drawing codes. Fortunately, the PDF drawing codes can be written in plain text, rather than binary, so I could edit what I wanted drawn with just a text editor.
The xref table and a few other things add a bit of complexity, because they have counts or offsets of other data in the file. If a plain text file is edited, the offsets and sizes change, and the PDF xref table and other things would be wrong. So as a practical matter, the structural elements like xref, objs, and streams are better produced by a program, while the content of the streams that actually do the drawing can be plain text.
As you can probably guess at this point, I wrote a simple script that takes lines of text representing PDF drawing commands, and combines them with the necessary structural parts into a valid PDF file. It turned out a bit more complex than that, because I did things incrementally, so there are multiple object streams that make up a page, even though it's just a single page per PDF file.
I also ended up eliminating most of the intermediate language, so the graph-maker stage produces a lot of PDF drawing commands directly, with only a couple of things like page-dimensions still in intermediate form.
I produce the statistics graphs (and my haunting avatars) in PDF, which I then convert to GIF using the 'sips' command.
The How
The input to the graph-maker stage is a series of lines containing the data for a single user on a single date. The data for each user is collected into a polyline, which means a series of line segments that make up a user's line graph. In addition, min and max values are calculated across all users and dates, and that data is used to automatically determine the extent of the final graph.
This data is then fused with some configurable (but fairly stable) drawing data, such as dimensions, colors, line-widths, font sizes, etc. and that's all then fed into the actual PDF-generator.
The whole production is scripted. It's a combination of shell scripts (bash), awk scripts, and commands like sort, head, tail, tr, sips, etc. The whole thing is automated, so all I do is tell the script to use new data, such as what I collected on 13 October, and it runs through it all, producing a couple dozen PDF files in about 10 seconds.
If I want to adjust a particular graph, I edit one text file that determines which users and which dates to select from all the collected data. Then I run the script to produce all the graphs again. Yes, it unnnecessarily repeats the production of 20 or so PDF files, but since it only takes 10 seconds, I don't much care.
Once the PDF graphs look the way I want, I then convert them to GIF files using the 'sips' command. This is also done in a script, because there are two distinct graph sizes, and 'sips' needs to be told what output size for each group.
Finally, I write up the thrilling descriptions, and put it all together in a post on MacRumors.
In general, unless I have a completely new graph, or significant changes to past ones, the actual production of the graphs takes only a minute or so, because it's almost completely automated. From selecting the Top 50, sorting by Likes, splitting into groups, or setting graph titles, it's all in different scripts, and all I need to do is run it. The most time-consuming part is looking at the graphs, picking out interesting things to remark on, and writing the text of the posts.
The Why
When I first looked into producing graphs, I considered a few ways to do it.
One of the criteria was that it be automatable, meaning I could take a bunch of data output from an awk program (or other script) and turn it into a line graph. I was mainly interested in line graphs, because they're a natural fit when the data is counts obtained on dates.
One approach would be to use a graphing or charting package, such as the ones listed here:
List of charting software - Wikipedia
I looked at a few, and downloaded a couple that offered compilable source, and built them. I also went through some user's guides, to see how to tell them what to do.
If I chose a charting package, even an automatable one, I'd have a lot of flexibility in picking different kinds of charts and graphs. Sadly, that flexibility wouldn't really be useful, because the most common graph I expected would pretty much always be a line graph.
I also considered what tools or knowledge I'd gain if I used an existing automated charting package. Basically, I'd know how to automate one charting package to produce line graphs. As useful as that would be for the purpose of producing graphs, that wasn't my only goal. To me, it seemed like a fairly narrow skill.
On the other hand, suppose I made a simple ad hoc graphing package of my own, which output to one of the image file formats:
Image file format - Wikipedia
It wouldn't need to be complex, with many different graph or chart types, because I'd be almost exclusively making line graphs.
I already knew that 'sips' can convert between several formats, so I could use a direct output format that was convenient for production, even if I uploaded a different format like GIF or PNG.
From what I already knew about PDF, it has a compact and easily represented way to draw polylines, set colors, and other simple drawing operations. It also has a lot of control over rendering text, which I knew I'd need for labels and such.
So if I wrote my own graph-maker that produced PDF, I'd learn a broader set of skills, namely, how to build simple line graphs from raw numbers, and how to draw things with PDF.
The What
The first stage was to turn a bunch of numbers, dates, and names into a halfway decent line graph. For that, I wrote scripts that produced output in a very simplified "intermediate language" that said things like:
LINE x1 y1 x2 y2
LABEL x1 y1 string
I was fairly confident that once I got the basic graph production working, this intermediate form could be replaced by a real PDF generator.
The second stage was to learn enough basic PDF to draw a few things on a page. That turned out to need a bit of "housekeeping" in order to make a well-formed PDF file.
It turns out that a PDF file has some required elements, such as an xref table and multiple objects with streams that contain the actual drawing codes. Fortunately, the PDF drawing codes can be written in plain text, rather than binary, so I could edit what I wanted drawn with just a text editor.
The xref table and a few other things add a bit of complexity, because they have counts or offsets of other data in the file. If a plain text file is edited, the offsets and sizes change, and the PDF xref table and other things would be wrong. So as a practical matter, the structural elements like xref, objs, and streams are better produced by a program, while the content of the streams that actually do the drawing can be plain text.
As you can probably guess at this point, I wrote a simple script that takes lines of text representing PDF drawing commands, and combines them with the necessary structural parts into a valid PDF file. It turned out a bit more complex than that, because I did things incrementally, so there are multiple object streams that make up a page, even though it's just a single page per PDF file.
I also ended up eliminating most of the intermediate language, so the graph-maker stage produces a lot of PDF drawing commands directly, with only a couple of things like page-dimensions still in intermediate form.
? ?
Prompted by a question from @SandboxGeneral in another thread, I realize I haven't described the How or Why behind the statistics graphs. I will now remedy that.
I produce the statistics graphs (and my haunting avatars) in PDF, which I then convert to GIF using the 'sips' command.
The How
The input to the graph-maker stage is a series of lines containing the data for a single user on a single date. The data for each user is collected into a polyline, which means a series of line segments that make up a user's line graph. In addition, min and max values are calculated across all users and dates, and that data is used to automatically determine the extent of the final graph.
This data is then fused with some configurable (but fairly stable) drawing data, such as dimensions, colors, line-widths, font sizes, etc. and that's all then fed into the actual PDF-generator.
The whole production is scripted. It's a combination of shell scripts (bash), awk scripts, and commands like sort, head, tail, tr, sips, etc. The whole thing is automated, so all I do is tell the script to use new data, such as what I collected on 13 October, and it runs through it all, producing a couple dozen PDF files in about 10 seconds.
If I want to adjust a particular graph, I edit one text file that determines which users and which dates to select from all the collected data. Then I run the script to produce all the graphs again. Yes, it unnnecessarily repeats the production of 20 or so PDF files, but since it only takes 10 seconds, I don't much care.
Once the PDF graphs look the way I want, I then convert them to GIF files using the 'sips' command. This is also done in a script, because there are two distinct graph sizes, and 'sips' needs to be told what output size for each group.
Finally, I write up the thrilling descriptions, and put it all together in a post on MacRumors.
In general, unless I have a completely new graph, or significant changes to past ones, the actual production of the graphs takes only a minute or so, because it's almost completely automated. From selecting the Top 50, sorting by Likes, splitting into groups, or setting graph titles, it's all in different scripts, and all I need to do is run it. The most time-consuming part is looking at the graphs, picking out interesting things to remark on, and writing the text of the posts.
The Why
When I first looked into producing graphs, I considered a few ways to do it.
One of the criteria was that it be automatable, meaning I could take a bunch of data output from an awk program (or other script) and turn it into a line graph. I was mainly interested in line graphs, because they're a natural fit when the data is counts obtained on dates.
One approach would be to use a graphing or charting package, such as the ones listed here:
List of charting software - Wikipedia
I looked at a few, and downloaded a couple that offered compilable source, and built them. I also went through some user's guides, to see how to tell them what to do.
If I chose a charting package, even an automatable one, I'd have a lot of flexibility in picking different kinds of charts and graphs. Sadly, that flexibility wouldn't really be useful, because the most common graph I expected would pretty much always be a line graph.
I also considered what tools or knowledge I'd gain if I used an existing automated charting package. Basically, I'd know how to automate one charting package to produce line graphs. As useful as that would be for the purpose of producing graphs, that wasn't my only goal. To me, it seemed like a fairly narrow skill.
On the other hand, suppose I made a simple ad hoc graphing package of my own, which output to one of the image file formats:
Image file format - Wikipedia
It wouldn't need to be complex, with many different graph or chart types, because I'd be almost exclusively making line graphs.
I already knew that 'sips' can convert between several formats, so I could use a direct output format that was convenient for production, even if I uploaded a different format like GIF or PNG.
From what I already knew about PDF, it has a compact and easily represented way to draw polylines, set colors, and other simple drawing operations. It also has a lot of control over rendering text, which I knew I'd need for labels and such.
So if I wrote my own graph-maker that produced PDF, I'd learn a broader set of skills, namely, how to build simple line graphs from raw numbers, and how to draw things with PDF.
The What
The first stage was to turn a bunch of numbers, dates, and names into a halfway decent line graph. For that, I wrote scripts that produced output in a very simplified "intermediate language" that said things like:
LINE x1 y1 x2 y2LABEL x1 y1 string
I was fairly confident that once I got the basic graph production working, this intermediate form could be replaced by a real PDF generator.
The second stage was to learn enough basic PDF to draw a few things on a page. That turned out to need a bit of "housekeeping" in order to make a well-formed PDF file.
It turns out that a PDF file has some required elements, such as an xref table and multiple objects with streams that contain the actual drawing codes. Fortunately, the PDF drawing codes can be written in plain text, rather than binary, so I could edit what I wanted drawn with just a text editor.
The xref table and a few other things add a bit of complexity, because they have counts or offsets of other data in the file. If a plain text file is edited, the offsets and sizes change, and the PDF xref table and other things would be wrong. So as a practical matter, the structural elements like xref, objs, and streams are better produced by a program, while the content of the streams that actually do the drawing can be plain text.
As you can probably guess at this point, I wrote a simple script that takes lines of text representing PDF drawing commands, and combines them with the necessary structural parts into a valid PDF file. It turned out a bit more complex than that, because I did things incrementally, so there are multiple object streams that make up a page, even though it's just a single page per PDF file.
I also ended up eliminating most of the intermediate language, so the graph-maker stage produces a lot of PDF drawing commands directly, with only a couple of things like page-dimensions still in intermediate form.
Way over my head and skill set. But fascinating thought process and delivery as always.Prompted by a question from @SandboxGeneral in another thread, I realize I haven't described the How or Why behind the statistics graphs. I will now remedy that.
I produce the statistics graphs (and my haunting avatars) in PDF, which I then convert to GIF using the 'sips' command.
The How
The input to the graph-maker stage is a series of lines containing the data for a single user on a single date. The data for each user is collected into a polyline, which means a series of line segments that make up a user's line graph. In addition, min and max values are calculated across all users and dates, and that data is used to automatically determine the extent of the final graph.
This data is then fused with some configurable (but fairly stable) drawing data, such as dimensions, colors, line-widths, font sizes, etc. and that's all then fed into the actual PDF-generator.
The whole production is scripted. It's a combination of shell scripts (bash), awk scripts, and commands like sort, head, tail, tr, sips, etc. The whole thing is automated, so all I do is tell the script to use new data, such as what I collected on 13 October, and it runs through it all, producing a couple dozen PDF files in about 10 seconds.
If I want to adjust a particular graph, I edit one text file that determines which users and which dates to select from all the collected data. Then I run the script to produce all the graphs again. Yes, it unnnecessarily repeats the production of 20 or so PDF files, but since it only takes 10 seconds, I don't much care.
Once the PDF graphs look the way I want, I then convert them to GIF files using the 'sips' command. This is also done in a script, because there are two distinct graph sizes, and 'sips' needs to be told what output size for each group.
Finally, I write up the thrilling descriptions, and put it all together in a post on MacRumors.
In general, unless I have a completely new graph, or significant changes to past ones, the actual production of the graphs takes only a minute or so, because it's almost completely automated. From selecting the Top 50, sorting by Likes, splitting into groups, or setting graph titles, it's all in different scripts, and all I need to do is run it. The most time-consuming part is looking at the graphs, picking out interesting things to remark on, and writing the text of the posts.
The Why
When I first looked into producing graphs, I considered a few ways to do it.
One of the criteria was that it be automatable, meaning I could take a bunch of data output from an awk program (or other script) and turn it into a line graph. I was mainly interested in line graphs, because they're a natural fit when the data is counts obtained on dates.
One approach would be to use a graphing or charting package, such as the ones listed here:
List of charting software - Wikipedia
I looked at a few, and downloaded a couple that offered compilable source, and built them. I also went through some user's guides, to see how to tell them what to do.
If I chose a charting package, even an automatable one, I'd have a lot of flexibility in picking different kinds of charts and graphs. Sadly, that flexibility wouldn't really be useful, because the most common graph I expected would pretty much always be a line graph.
I also considered what tools or knowledge I'd gain if I used an existing automated charting package. Basically, I'd know how to automate one charting package to produce line graphs. As useful as that would be for the purpose of producing graphs, that wasn't my only goal. To me, it seemed like a fairly narrow skill.
On the other hand, suppose I made a simple ad hoc graphing package of my own, which output to one of the image file formats:
Image file format - Wikipedia
It wouldn't need to be complex, with many different graph or chart types, because I'd be almost exclusively making line graphs.
I already knew that 'sips' can convert between several formats, so I could use a direct output format that was convenient for production, even if I uploaded a different format like GIF or PNG.
From what I already knew about PDF, it has a compact and easily represented way to draw polylines, set colors, and other simple drawing operations. It also has a lot of control over rendering text, which I knew I'd need for labels and such.
So if I wrote my own graph-maker that produced PDF, I'd learn a broader set of skills, namely, how to build simple line graphs from raw numbers, and how to draw things with PDF.
The What
The first stage was to turn a bunch of numbers, dates, and names into a halfway decent line graph. For that, I wrote scripts that produced output in a very simplified "intermediate language" that said things like:
LINE x1 y1 x2 y2LABEL x1 y1 string
I was fairly confident that once I got the basic graph production working, this intermediate form could be replaced by a real PDF generator.
The second stage was to learn enough basic PDF to draw a few things on a page. That turned out to need a bit of "housekeeping" in order to make a well-formed PDF file.
It turns out that a PDF file has some required elements, such as an xref table and multiple objects with streams that contain the actual drawing codes. Fortunately, the PDF drawing codes can be written in plain text, rather than binary, so I could edit what I wanted drawn with just a text editor.
The xref table and a few other things add a bit of complexity, because they have counts or offsets of other data in the file. If a plain text file is edited, the offsets and sizes change, and the PDF xref table and other things would be wrong. So as a practical matter, the structural elements like xref, objs, and streams are better produced by a program, while the content of the streams that actually do the drawing can be plain text.
As you can probably guess at this point, I wrote a simple script that takes lines of text representing PDF drawing commands, and combines them with the necessary structural parts into a valid PDF file. It turned out a bit more complex than that, because I did things incrementally, so there are multiple object streams that make up a page, even though it's just a single page per PDF file.
I also ended up eliminating most of the intermediate language, so the graph-maker stage produces a lot of PDF drawing commands directly, with only a couple of things like page-dimensions still in intermediate form.