Is Ignite a persistent or pure in-memory storage?

Is Ignite an in-memory database (IMDB)?

Is Ignite an in-memory data grid (IMDG)?

Is Ignite a distributed database?

Is Ignite an SQL database?

Is Ignite a transactional database? 

Is Ignite a key-value store?

1. Durable Memory

Ignite durable memory component treats RAM not just as a caching layer, but as a complete fully-functional storage layer. This means that users can turn the persistence on and off as needed. If the persistence is off, then Ignite, just like always, can act as a distributed In-Memory Database or as an In-Memory Data Grid, depending whether you prefer to use SQL or key-value APIs. If the persistence is turned on, then Ignite becomes a distributed, horizontally scalable database that guarantees full data consistency and is resilient to full cluster failures. On top of that, the data is stored in off-heap memory so there are no GC pauses even on large data sets.

2. Complete SQL support

3. ACID compliance

4. Collocated Processing

Most traditional SQL and NoSQL databases work in a client-server fashion, meaning that data must be brought to the client side for processing. This approach requires lots of data movement from servers to clients and generally does not scale.  Ignite, on the other hand, allows for sending computations to the data, moving only the light weight compute functions across the network. As a result, Ignite scales better and minimizes data movement. When collocated, all the data processing happens locally on the node that stores the data, and only the result is brought back to the user.

5. Scalability and Durability

Deadlock-Free Transactions

Zero-Deployment (revisited)

Pushing Performance Boundaries

Yardstick Framework

Transparency

Hardware

Benchmarks

try (Transaction tx = ignite().transactions().txStart()) {
    Object val = cache.get(key);

    if (val != null)
        key = nextRandom(args.range() / 2, args.range());

    cache.put(key, new SampleValue(key));

    tx.commit();
}

TransactionContext tCtx = hazelcast().newTransactionContext(txOpts);

tCtx.beginTransaction();

TransactionalMap<Object, Object> txMap = tCtx.getMap("map");

Object val = txMap.getForUpdate(key);

if (val != null)
    key = nextRandom(args.range() / 2, args.range());

txMap.put(key, new SampleValue(key));

tCtx.commitTransaction();

Just like with optimistic transactions, we found that in pessimistic mode, Apache Ignite data grid is about 44% faster than Hazelcast. Apache Ignite averaged approximately 16,500 transactions per second, while Hazelcast came in at about 11,000 transactions per second.

Here is a sample graph produced by Yardstick:

Also, when running Hazelcast benchmarks, the following exception kept popping up in the logs, which keeps me wondering about the consistency of the data cached in Hazelcast overall:

SEVERE: [172.30.1.95]:57500 [dev] [3.4.2] Lock is not owned by the transaction! Caller: fa705359-7154-4346-a5f2-292e1a2a75a5, Owner: Owner: fa705359-7154-4346-a5f2-292e1a2a75a5, thread-id: 105
com.hazelcast.transaction.TransactionException: Lock is not owned by the transaction! Caller: fa705359-7154-4346-a5f2-292e1a2a75a5, Owner: Owner: fa705359-7154-4346-a5f2-292e1a2a75a5, thread-id: 105
        at com.hazelcast.map.impl.tx.TxnPrepareBackupOperation.run(TxnPrepareBackupOperation.java:48)
        at com.hazelcast.spi.impl.Backup.run(Backup.java:92)
        at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.handle(BasicOperationService.java:749)
        at com.hazelcast.spi.impl.BasicOperationService$OperationHandler.access$500(BasicOperationService.java:725)
        at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:699)
        at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.handle(BasicOperationService.java:643)
        at com.hazelcast.spi.impl.BasicOperationService$OperationPacketHandler.access$1500(BasicOperationService.java:630)
        at com.hazelcast.spi.impl.BasicOperationService$BasicDispatcherImpl.dispatch(BasicOperationService.java:582)
        at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.process(BasicOperationScheduler.java:466)
        at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.doRun(BasicOperationScheduler.java:458)
        at com.hazelcast.spi.impl.BasicOperationScheduler$OperationThread.run(BasicOperationScheduler.java:432)

Yardstick Framework

Transparency

Hardware

Benchmarks

We compared basic puts and puts-and-gets into the cache.

The code used for the benchmark execution can be found on GitHub:

• Apache Ignite: IgnitePutBenchmark and IgnitePutGetBenchmark.
• Hazelcast: HazelcastPutBenchmark and HazelcastPutGetBenchmark.

Result:
We found that both Ignite and Hazelcast exhibit about the same performance with Ignite being about 4% to 7% faster on most of the runs.

Here are the graphs produced by Yardstick:

Basic Transaction Operations

We compared basic transactional puts and puts-and-gets into the cache in OPTIMISTIC mode.

The code used for the benchmark execution can be found on GitHub:

• Apache Ignite: IgnitePutTxBenchmark and IgnitePutGetTxBenchmark.
• Hazelcast: HazelcastPutTxBenchmark and HazelcastPutGetTxBenchmark.

Result:
The performance difference for OPTIMISTIC transactions was much bigger, with Ignite transactions outperforming Hazelcast transactions by about 35% to 45%.

Here are the graphs produced by Yardstick:

1. The stream gets partitioned to multiple cluster nodes in such a way that same numbers will always be processed on the same node. 
2. Upon receiving a number, our StreamTransformer will get the current count for that number and increment it by 1.

try (IgniteDataStreamer<Integer, Long> stmr = ignite.dataStreamer("numbers")) {
    // Allow data updates.
    stmr.allowOverwrite(true);

    // Configure data transformation to count random numbers 
    // added to the stream.
    stmr.receiver(StreamTransformer.from((e, arg) -> {
        // Get current count.
        Long val = e.getValue();

        // Increment count by 1.
        e.setValue(val == null ? 1L : val + 1);

        return null;
    }));

    // Stream 10 million of random numbers in the range of 0 to 1000.
    for (int i = 1; i <= 10_000_000; i++) {
        stmr.addData(RAND.nextInt(1000), 1L);

        if (i % 500_000 == 0)
            System.out.println("Number of tuples streamed into Ignite: " + i);
    }
}

// Query top 10 most popular numbers every.
SqlFieldsQuery top10Qry = new SqlFieldsQuery(
    "select _key, _val from Long order by _val desc limit 10");

// Execute query and get the whole result set.
List<List<?>> top10 = stmCache.query(top10Qry).getAll();

1. We will setup up a cache to hold the words as they come from a stream.
2. We will setup a 1 second sliding window to keep the words only for the last 1 second.
3. StreamWords program will stream text data into Ignite.
4. QueryWords program will query top 10 words out of the stream.

Cache Configuration

public class CacheConfig {
  public static CacheConfiguration<String, Long> wordCache() {
    CacheConfiguration<String, Long> cfg = new CacheConfiguration<>("words");

    // Index individual words.
    cfg.setIndexedTypes(AffinityUuid.class, /*word type*/String.class);

    // Sliding window of 1 seconds.
    cfg.setExpiryPolicyFactory(FactoryBuilder.factoryOf(
      new CreatedExpiryPolicy(new Duration(SECONDS, 1))));

    return cfg;
  }
}

Stream Words

public class StreamWords {
  public static void main(String[] args) throws Exception {
    // Mark this cluster member as client.
    Ignition.setClientMode(true);

    try (Ignite ignite = Ignition.start("examples/config/example-ignite.xml")) {
      // The cache is configured with sliding window holding 1 second of the streaming data.
      IgniteCache<AffinityUuid, String> stmCache = ignite.getOrCreateCache(CacheConfig.wordCache());

      try (IgniteDataStreamer<AffinityUuid, String> stmr = ignite.dataStreamer(stmCache.getName())) {
        // Stream words from "alice-in-wonderland" book.
        while (true) {
          InputStream in = StreamWords.class.getResourceAsStream("alice-in-wonderland.txt");

          try (LineNumberReader rdr = new LineNumberReader(new InputStreamReader(in))) {
            for (String line = rdr.readLine(); line != null; line = rdr.readLine()) {
              for (String word : line.split(" "))
                if (!word.isEmpty())
                  // Stream words into Ignite.
                  // By using AffinityUuid as a key, we ensure that identical
                  // words are processed on the same cluster node.
                  stmr.addData(new AffinityUuid(word), word);
            }
          }
        }
      }
    }
  }
}

Query Words

1. We use standard SQL to query popular words.
2. Ignite SQL treats Java classes as SQL tables. Since our words are stored as simple String type, the SQL query below queries String table.
3. Ignite always stores cache keys and values as "_key" and "_val" fields. In our case, "_val" is the word, so we use this syntax in our SQL query.

public class QueryWords {
  public static void main(String[] args) throws Exception {
    // Mark this cluster member as client.
    Ignition.setClientMode(true);

    try (Ignite ignite = Ignition.start()) {
      IgniteCache<String, Long> stmCache = ignite.getOrCreateCache(CacheConfig.wordCache());

      // Select top 10 words.
      SqlFieldsQuery top10Qry = new SqlFieldsQuery(
          "select _val, count(_val) as cnt from String " + 
            "group by _val " + 
            "order by cnt desc " + 
            "limit 10",
          true /*collocated*/
      );

      // Query top 10 popular numbers every 5 seconds.
      while (true) {
        // Execute queries.
        List<List<?>> top10 = stmCache.query(top10Qry).getAll();

        // Print top 10 words.
        ExamplesUtils.printQueryResults(top10);

        Thread.sleep(5000);
      }
    }
  }
}

Starting Server Nodes

bin/ignite.sh

public class ExampleNodeStartup {
    public static void main(String[] args) throws IgniteException {
        Ignition.start();
    }
}

...
Query results:
(the,2890)
(and,1355)
(to,1298)
(a,1139)
(of,1029)
(said,1002)
(in,912)
(she,820)
(was,766)
(you,711)
Query results:
(the,1679)
(to,830)
(and,810)
(a,680)
(of,629)
(she,491)
(it,357)
(in,330)
(said,315)
(was,274)
...

Enhance the Java Futures

public interface IgniteFuture<V> extends Future<V> {
    ...

    /**
     * Registers listener closure to be asynchronously notified whenever future completes.
     *
     * @param lsnr Listener closure to register. If not provided - this method is no-op.
     */
    public void listen(IgniteInClosure<? super IgniteFuture<V>> lsnr);

    /**
     * Make a chained future to convert result of this future (when complete) into a new format.
     * It is guaranteed that done callback will be called only ONCE.
     *
     * @param doneCb Done callback that is applied to this future when it finishes to produce chained future result.
     * @return Chained future that finishes after this future completes and done callback is called.
     */
    public <T> IgniteFuture<T> chain(IgniteClosure<? super IgniteFuture<V>, T> doneCb);
    
    ...
}

IgniteAsyncSupport

// IgniteCompute has synchronous and asynchronous modes.
IgniteCompute compute = ignite.compute();

// Execute a job synchronously and wait for the result.
String res = compute.call(() -> "Hello world");

// Enable asynchronous mode (note that the same IgniteCompute API is used).
IgniteCompute asyncCompute = ignite.compute().withAsync();

// Asynchronously execute a job.
asyncCompute.call(() -> "Hello world");

// Get the future for the above invocation.
IgniteFuture<String> fut = asyncCompute.future();

// Asynchronously listen for completion and print out the result.
fut.listen(f -> System.out.println("Job result: " + f.get()));

IgniteCache<String, Integer> cache = ignite.jcache("mycache");

// Synchronously store value in cache and get previous value.
Integer val = cache.getAndPut("1", 1);

// Enable asynchronous mode (note that the same IgniteCache API is used).
IgniteCache<String, Integer> asyncCache = ignite.jcache("mycache").withAsync();

// Asynchronously store value in cache.
asyncCache.getAndPut("1", 1);

// Get future for the above invocation.
IgniteFuture<Integer> fut = asyncCache.future();

// Asynchronously listen for the operation to complete.
fut.listenAsync(f -> System.out.println("Previous cache value: " + f.get()));

What is a Data Grid

• distributed transactions
• distributed queries
• collocation of compute and data

• SQL support
• Off-Heap Memory (to avoid lengthy GC pauses)
• WebSession Caching
• Hibernate Integration
• Database Integration 

What is an In-Memory Data Fabric

• Data Grid (must have for any Data Fabric)
• Compute Grid
• Service Grid
• Streaming & CEP
• Distributed File System
• In-Memory Database