Conversation

Notices

1. musicman (musicman@nu.federati.net)'s status on Thursday, 29-Oct-2020 10:26:01 EDT musicman
   This seems like a lot: "store up to 2 years worth of metrics."
   
   I guess it depends how much data is being collected, so that's what I need to figure out.
   
   Anybody collecting metrics? How much do you store?
   1. lnxw48a1 (lnxw48a1@nu.federati.net)'s status on Thursday, 29-Oct-2020 11:55:07 EDT lnxw48a1

      in reply to
      @musicman I don't, but even simple logs can consume a surprising amount of storage. There is a reason why logrorate is needed.
      1. GeniusMusing (geniusmusing@nu.federati.net)'s status on Friday, 30-Oct-2020 21:30:18 EDT GeniusMusing

         in reply to
         @musicman @lnxw48a1
         My backup logs for October are ~124MB
         Two years would be about 3GB.
         I only keep them for about a month and I could probably dump them as they are scanned as error free.
         1. musicman (musicman@nu.federati.net)'s status on Monday, 02-Nov-2020 08:46:17 EST musicman

            in reply to
            how many machines does that cover? In this case, we're talking about everything #Prometheus collects.
            1. GeniusMusing (geniusmusing@nu.federati.net)'s status on Monday, 02-Nov-2020 10:10:49 EST GeniusMusing

               in reply to
               @musicman My backup (rsync) logs are for one physical machine, 2 VM's, 6 shares, and about 600GB worth of files. This is my home/work/company server.
               1. musicman (musicman@nu.federati.net)'s status on Monday, 02-Nov-2020 16:11:33 EST musicman

                  in reply to
                  I don't actually know who this is because it is coming through a partner, but I suspect we are talking more data. I guess I will see.
                  1. GeniusMusing (geniusmusing@nu.federati.net)'s status on Monday, 02-Nov-2020 16:58:58 EST GeniusMusing

                     in reply to
                     @musicman Not having used Prometheus before I don't have any idea but I did find this, not sure if it is relevant.
                     
                     Prometheus monitoring at scale with the Elastic Stack Elastic Blog
                     https://www.elastic.co/blog/prometheus-monitoring-at-scale-with-the-elastic-stack
                     
                     >Long-term data retention
                     >
                     >Prometheus stores data locally within the instance. Having both compute and data storage on one node may make it easier to operate, but also makes it harder to scale and ensure high availability. As a consequence, Prometheus is not optimized to be a long-term metrics store. Depending on the size of your environment your optimal retention rate for time series in Prometheus can be as short as several days or even hours.