GridGain 3.6.0 Released!

I am glad to announce that we have released GridGain 3.6.0 - Real Time Big Data platform that allows anyone easily develop, scale and manage compute and data intensive big data applications using integrated Compute and In-Memory Data Grids with Java, Scala and Groovy native APIs.

Some of the key new features and enhancements:

• “Write From Behind Caching” was finally added to IMDGs
• REST batch operation support introduced (putAll, removeAll)
• Eviction filters added
• Segmentation handling for half-connected network sockets
• Improvements in GridGain Visor
• New eviction performance optimizations
• Significant IMDG preloading performance enhancements
• New optimization for lock-free reads and low contention writes
• Cumulative bug fixes and multitude of performance

You can download it here.

GridGain is hiring

GridGain Systems is rapidly growing and we are hiring talented Java/Scala top-level engineers in both of our locations:
- Saint Petersburg, Russia
- Foster City, CA, USA

Come work for one of the hotest startups in the Silicon Valley! Cutting edge work and top level benefits and compensation. We also guarantee sleepless nights, unsolvable problems and insane debugging of complex distributed systems just in the first week 

All the details are here.

GridGain 3.5 Released!

We’ve just released GridGain 3.5 – our latest stable release of GridGain’s Real Time Big Data platform that allows anyone easily develop, scale and manage compute and data intensive JVM based applications using integrated Compute and In-Memory Data Grid with Java, Scala and Groovy native APIs.

Some of the key new features and enhancements:

• Significant performance improvements throughout the product 
• Previously deprecated APIs have been removed 
• Improvements and bug fixes in GridGain Visor 
• Enhancements to GridProjection interface 
• Ability to programmatically start remote nodes in bulk 
• Customizable closure-based MapReduce 
• Enhancements in affinity and co-location 
• Bug fixes in eviction policies in Data Grid 
• Significant improvements and performance enhancements in Swap SPI 

Download  GridGain 3.5 today!

PDC Theorem – The Underlying Principle of Real-Time Distributed Processing

Nikita, CEO at GridGain Systems, wrote a cool blog on PDC Theorem – The Underlying Principle of Real-Time Distributed Processing where he talks in some detail about important principles of distributed programming, such as real-time processing, data partitioning, and collocation of computations and data. Take a look at it - should be a good and interesting read.

How about distributed queues?

Did you ever wish you could take a data structure you are familiar with and distribute it over grid? For example, why not take java.util.concurrent.BlockingDeque and add something to it on one node and poll it from another node? Or why not have a distributed primary key generator which would guarantee uniqueness on all nodes? Or how about a distributed java.util.concurrent.atomic.AtomicLong which can be updated and read from any node on the grid? GridGain gives you such capability. What GridGain did is actually take most of the data structures from java.util.concurrent framework and made sure they could be used in distributed fashion.

In this blog I want to show how flexible GridGain distributed queues are. On top implementing java.util.Collection interface and supporting different modes of operation, like collocated vs. non-collocated, or bounded vs. ubounded modes, you can actually control how elements are ordered within queues. GridGain supports FIFO, LIFO, and Priority based queues out of the box.

FIFO queues (first-in-first-out) are the most traditional queues where elements are added from the tail and polled form the queue head. LIFO queues (last-in-first-out) resemble more of stack features instead of queues. In LIFO queues elements are added and polled from the tail.

But the most interesting queue type is Priority queue where user can control the order of the elements. Priority queues order elements within the queue based on priority attribute specified by the user. Priority attribute of a queue element is annotated via @GridCacheQueuePriority annotation. Here is an example of how priority queue can be created and used.

public void priorityQueueExample() {
    Random rand = new Random();

    Grid grid = G.grid();

    // Initialize new unbounded collocated priority queue.
    GridCacheQueue<PriorityItem> queue = 
        grid.cache().queue("myqueue", PRIORITY);

    // Store 20 elements in queue with random priority.
    for (int i = 0; i < 20; i++) {
        int priority = rand.nextInt(20);

        queue.put(new PriorityItem(priority, "somedata-" + i));
    }

    PriorityItem item = null;

    int lastPriority = 0;

    do {
        item = queue.poll();

        // Ensure the elements are correctly ordered based on priority.
        assert lastPriority <= item.priority();

        lastPriority = item.priority();
    }
    while (item != null);
}

...

// Class defining sample queue element with its priority specified via
// @GridCacheQueuePriority annotation attached to priority field.
private static class PriorityItem implements Serializable {
    // Priority of queue item.
    @GridCacheQueuePriority
    private final int priority;

    private final String data;

    private SampleItem(int priority, String data) {
        this.priority = priority;
        this.data = data;
    }

    public int priority() {
        return priority;
    }
}

Read more about GridGain queues here.

Clever Singletons on the Grid

When working in distributed environment often you need to have a consistent local state per grid node that is reused between various job executions. For example, what if multiple jobs require database connection pool for their execution - how do they get this connection pool to be initialized once and then reused by all jobs running on the same grid node? Essentially you can think about it as a per-grid-node singleton service, but the idea is not limited to services only, it can be just a regular Java bean that holds some state to be shared by all jobs running on the same grid node.

In GridGain versions 2.x and earlier this approach was handled by using @GridUserResource annotation to annotate fields within GridTask or GridJob classes to specify singleton beans. However, this approach was dependent on GridDeploymentMode configuration and, for ISOLATED or PRIVATE deployment modes, resource could be initialized multiple times, once per GridTask. This forced users to use various hacks in their logic and generally was not very convenient to use.

Starting with GridGain 3.0, GridNodeLocal per-grid-node local storage was introduced. The name was borrowed from ThreadLocal class in Java, because just like ThreadLocal provides unique per-thread space in Java, GridNodeLocal provides unique per-grid-node space in GridGain. GridNodeLocal implements java.util.concurrent.ConcurrentMap interface and is absolutely lock-free. In fact, it simply extends java.util.concurrent.ConcurrentHashMap implementation and, therefore, inherits all the methods available there.

Here is an example of how GridNodeLocal could be used to create some user specific singleton service from a simple GridGain job:

final Grid grid = G.start(..);

// Execute runnable job on some remote grid node.
grid.run(BALANCE, new Runnable() {
  public void run() {
    GridNodeLocal<String, MySingleton> nodeLocal = grid.nodeLocal();

    // 1. Check if f someone already created our singleton service. 
    MySingleton service = nodeLocal.get("myservice");

    if (service == null) {
      // 2. Create new singleton service.
      MySingleton other = 
        nodeLocal.putIfAbsent("myservice", service = new MySingleton(..));

      if (other != null) 
        service = other;
    }

    // Perform operations with our singleton service.
    ...
  }
});

Even better AtomicInteger and AtomicLong in Java

Last week I blogged about "Better AtomicInteger and AtomicLong in Java" and how I added it to GridGain 'lang.utils' package to be used within GridGain project and by our user base. Recently I made it even better. The standard JDK atomic 'compareAndSet(..)' operations always check for some 'expected' value and if it matches, then the set happens. What if you simply want to set a value if its greater or less than current value, or if it simply does not match the current value?

Again, code looks pretty simple:

public class MyAtomicInteger extends AtomicInteger {
  ... // Omitting methods from previous blog
  public boolean setIfGreater(int update) {         
     while (true) {             
          int cur = get();

          if (update > cur) {
              if (compareAndSet(cur, update))
                  return true;
          }
          else
              return false;
      }
  }

  public boolean setIfGreaterEquals(int update) {
      while (true) {
          int cur = get();

          if (update >= cur) {
              if (compareAndSet(cur, update))
                  return true;
          }
          else
              return false;
      }
  }

  public boolean setIfLess(int update) {
      while (true) {
          int cur = get();
          
          if (update < cur) {
              if (compareAndSet(cur, update))
                 return true;
          }
          else
              return false;
      }
  }      
  
  public boolean setIfLessEquals(int update) {
      while (true) {
          int cur = get();

          if (update <= cur) {
              if (compareAndSet(cur, update))
                  return true;
          }
          else
              return false;
      }
  }

  public boolean setIfNotEquals(int update) {
      while (true) {
        int cur = get();

        if (update != cur) {
            if (compareAndSet(cur, update))
                return true;
        }
        else
            return false;
      }
  }
}