A recent article covered the ‘difficulties faced by lawyers searching large amounts of data under time pressure ’ – this is an increasingly common issue for many organisations not just lawyers.

Consider:

• Banks running out of time to calculate their capital adequacy during the working day.
• Passport controls needing to check details against ‘watch lists’ in real time.
• Forensic data investigators finding that the volumes make their task impossible.
• Professionals knowing what they are looking for but being unable to locate the information in reasonable time.

The common thread running through these problems is ‘too much data, too little time’. So what’s the problem?

Limitations of computers

Traditional data mining and analytical search tools are great at finding anything that can be described well with rules. They will thoroughly work through all the data looking for those pieces of information that match the criteria rules they’ve been set. In other words they are excellent at finding what you know is buried in the data you are searching.

Consider ‘looking for a needle in haystack’. This is easy as we know what the needle looks like – straight, shiny, metal, less than 10cm in length, etc. And we know what hay looks like. But what if the needle has been deliberately disguised? We know it is there but we are not certain what it now looks like or how it has been camouflaged.

How do traditional methods cope with the ‘too much data, too little time’ issue?

The usual solutions are:

• To cull the data to reasonable levels:
• Decide which data isn’t needed and throw some away.
• Stop collecting data from areas we aren’t interested in.

But how do we really know what we won’t need to reference in the future?

• To harness more technology:
• Purchase more computer power to solve the problem.
• Write complex algorithms to sort data more effectively.

Expensive solutions which invariably run out of steam again!

Which makes one wonder...

At some point - when your system is becoming ever more complex and you are twiddling your thumbs waiting for the results - you’re going to think “there must be a better way to do this”.

What if instead of reducing the data to get information you got better results with more data?

Or if the system could identify interesting data characteristics even though you hadn’t built it into the rules?

In fact what if you could get a computer system to look at all the data in the same way that a human does:

• Looking for interesting patterns
• Learning based on previous experience
• Becoming more certain of the results as more data is assimilated.

A better way – the Uncertainty Engine©

Sinus Iridum have developed the Uncertainty Engine© (UE) to take any data space and produce a multidimensional binary structure of up to 4000 elements in each of up to 4000 dimensions.

Just like the human brain - the UE:

• Reads in the data.
• Learns about the data patterns.
• Remembers the ‘pattern rules’ in computer memory in a very compact form.
• Considers all the positive and negative assertions before making a decision.
• Improves the accuracy of prediction through additional data.
• In decision making it retains information in an uncertain state until such time as it is certain.

Each additional piece of data to arrive updates all the previous knowledge giving the engine its real time performance and highly efficient, high 90’s predictive capability even with small amounts of input data.

Questions of the data can be posed in an English format such as ‘tell me what’s interesting’.

This structure of remaining uncertain, finding the patterns then verifying the patterns and finally becoming certain is in stark contrast to conventional computing that starts with a computer that has to be certain (0 or 1) from the outset and then giving it rules that are already known.

The Uncertainty Engine© can process data as quickly as it can be passed to it so the throughput is normally only limited by the size of the ‘pipe’ connecting the UE to the source data. This means that massive volumes of data can be processed in real-time giving the immediate system feedback that is often lacking in traditional analytical and data mining solutions.

So yes there is:

• A better way to analyse data
• A solution that gets around the limitations of many computer systems

A technique that doesn’t require lots of the data to be discarded before it starts