• Type: Sub-task
    • Status: Closed (View Workflow)
    • Priority: Major
    • Resolution: Done
    • Affects Version/s: None
    • Fix Version/s: 0.20.0
    • Component/s: Core
    • Labels:


      Job scheduling

      • * Create a job that is run every two hours
        • Timestamp block starts in even hours (00, 02, 04, ..)
        • Job runs on odd hours (01, 03, 05, ..)
      • It reads all the rows of the previous timestamp block (for example, 03 job reads everything written between 00-02)
        • Creates a compressed block
        • Writes it to the compressed table
        • Deletes the originals from data table

      Write path

      • Timestamp precision changes for the write path, can be done in the compression phase
        • Such as second precision instead of millisecond precision
          • Depending on the implementation, this needs to be unparsed in the read path - or then we just mark milliseconds as zero and be happy with that (in case of second precision)
      • Create wrapper for the gorilla-tsc library to add compression headers and to read them:
        • Type of compression that is used (1 byte) (bit mask?), allow different timestamp + value compression bundles
      • Create modifications to DataAccessImpl to write to the c_value column in data table and delete the selected keys from data
      • Since the writes and deletes go to the same partition always, the delete and insert can be done in a single batch -> performance improvements and no data loss
      • Out-of-order writes happen as before

      Other data types than Gauges

      • Create modifications to gorilla compressor to allow long compression also, not just double
        • This needs to be known in the decompressing.. set a bit in the header?
      • Write availabilities with current byte information (benefits from XOR compression also)
          • Fix the bug in the compressor that does not allow >= for leadingZeros

        Gliffy Diagrams




              • Assignee:
                burmanm Michael Burman
                burmanm Michael Burman
              • Votes:
                0 Vote for this issue
                1 Start watching this issue


                • Created: