• unskilled5117@feddit.org
    link
    fedilink
    English
    arrow-up
    20
    ·
    edit-2
    1 day ago

    That seems highly unlikely to me. Could a reason be website scrapers for AI using different user agents to prevent being blocked? The recent reports of different projects plagued by scrapers fit the timeline

      • unskilled5117@feddit.org
        link
        fedilink
        English
        arrow-up
        2
        ·
        edit-2
        17 hours ago

        Yeah i read that too. But how well is it working? I mean that’s what the news was all about the last few months. A lot off projects were having trouble blocking the bots because they were trying and succeeded to circumvent detection measures.

        • highball@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          1 hour ago

          But how well is it working?

          Yeah, that’s a good question. I don’t remember reading any documentation from Statcounter speaking about how well that bot detection and removal algorithms work. The last company I worked for did data collection, as we recorded all kinds of metrics from users. We had a team that focused on bot detection. Mostly it involved discovering the bot net and then removing all of their metrics that were accounted for in the system. Then updating the bloom filters so that any further metrics from the known bots would be ignored. Companies were paying millions of dollars for our contracts and every click was getting itemized and charged for. So, trust me, it was very important for my old company to discover and eliminate bots aggressively.

          Statcounters sells their statistics and not the data from what I understand. So it does beg the question, how aggressive are they?!