Cogniware Data Collector Core is the heart of the system managing Connectors and Data Handlers by providing an extensible enterprise level configuration mechanism. Configurations such as Connector connection credentials, Crawler query configuration, scheduling information and Data Handler connection information are stored in the product built-in database.

The framework allows multiple parallel crawling sessions for different queries, where jobs can be defined as full crawl jobs (for initial crawling) or incremental jobs (when scheduled or run manually).

This approach enables collecting of structured and unstructured data such as social network posts, comments, user profiles or web pages, as separate entities while taking into account the relations between them. These relations are also stored as separate objects and have similar structure to the mapping table in relational database.

Connectors estabilish connection to source systems. For each particular social media, file system, database, Sharepoint Site, etc. a separate connector needs to be created. Typical information that needs to be provided is a URL or IP Address of the source system and credentials to connect.

Data Handlers handle where the data retrieved by Connector will be stored (and analyzed). These are typically internal systems like IBM Watson Explorer or Cogniware GDPR Explorer or it can be Filesystem or SQL Database.

Crawlers are jobs defined by users and serve as definitions for crawling processes. Each crawler performs one crawling job and consists of connector, data handler and query definition.

Cogniware Data Collector User Interface is an HTML 5 WEB application. Users can use it to install and configure Connectors and Data Handlers and to create and manage Crawlers.