[geeks] RANT (ftp)

[Bryan Fullerton]

On 10/25/06, Charles Shannon Hendrix <shannon at widomaker.com> wrote:
> On Wed, 25 Oct 2006 10:32:30 -0400
> "Bryan Fullerton" <fehwalker at gmail.com> wrote:
>
> > I would have given a kidney to switch to Urchin back when I had to
> > support a WebTrends Enterprise install processing *log files* for a
> > 1M+ view/day web site (there is no logic in this, don't try to find
> > it).
>
> Just curious, what would you rather do than process log files to get
> the information?

Page tagging is the recommended way to use current "enterprise" stats
programs -- embed a reference to a 1x1 image in the page using
javascript, and stuff a whole bunch of crap about the client browser
into a cookie attached to that image. This is the way Google Analytics
works, and all the other big ASP stats providers.

You're still generating logs, but only one log line per page view,
which in our case would've saved around a terabyte of storage[1]. When
I left they were generating over 10Gb of bzip2 compressed logs per
week (including Akamai logs) under standard conditions, much more when
there were big news events.

Bryan

[1] Admittedly related to the mandated log file retention policy -
"keep them forever, and keep them available in case we want to do
adhoc historical reporting". They liked their stats a lot.