Are you curious about how long it takes for MadKudu to score your records? Look no further than the Processes page! As an Admin or Architect user, you can access this page to see when MadKudu last processed and scored your data in batch, as well as when the next batch update is scheduled. If you're interested in learning more about the different steps of the batch data pipeline and scoring process, this article is for you.

Prerequisites

• Page only available to Admin and Architect users

The Processes page tool (Settings > Processes) helps you understand when MadKudu is processing and scoring your data, and helps you see if MadKudu encountered any issue between the pull of your data, all the way to the sync (push) of the scores to your destinations in batch.

Please refer to this article to understand the difference between batch and real-time scoring.

When was the last time MadKudu scored my records in batch?

The last success of the sync process indicates when your records were last scored by the batch update.

When is the next batch update?

MadKudu processes your data in a specific order to send back scores to your integration. 

The data pipeline is executed continuously and usually runs end to end within 4h to 12h, or more depending on

• the volume of data (leads, contacts, accounts, events) from your integrations to process
• the number of models and the complexity of customizations to process. 

There are different reasons why a process may be failing or delayed (pipeline congested, server unavailable, connection interrupted ...etc)  but the batch will retry continuously to process the steps.  

Processes

The different steps of the batch data pipeline and scoring process are in the following order

1. merge: pulls data from your systems integrated into our platform (see Integrations tab) and import them into our data warehouse 
2. map: execute the Event mapping, Audience mapping and Conversion mapping
3. augment: enrich leads, contacts, and accounts with demographic, firmographic, technographic Enrichment from MadKudu's partners (Clearbit, HG, PredictLeads)
4. compute: execute the Attribute mapping, gathering all the additional enrichment available in your CRM and other integrations if any.
5. aggregations: execute the Aggregations (for LTB model)
6. analysis: execute all the live models configured for your organization to score leads, contacts, and accounts. 
7. sync: push the scores to your integrations (by batches)

Views and Janitor are additional processes for cleaning up our data warehouse.

Below is the representation of this data pipeline executed to transform your data into predictions or segmentations back to your CRM (or other systems) 

Last success

The last success column indicates when was the last time this process has run. 

The last success of the Sync process indicates when was the last time MadKudu has tried to score records in your systems. 

Status

The different statuses can be 

• idle: not running at the moment, waiting for its turn to process
• processing: currently being executed 
• queued: process in stand-by 
• failed: an error occurred, the process was not fully executed
• processing (last error: ): last try failed, the process is being executed at the moment

Process error codes

• worker shutdown: the server running the process on MadKudu side was momentarily down
  
  • Resolution: The process will retry by itself. 
• 1023 or 1018: several queries processing your data are trying to run at the same time and competing for the same resources
  • Resolution: The process will retry by itself. 
• 1040: the database where your data is processed is full
  • Resolution: The process will retry by itself. 
• Query cancelled on user's request: query processing your data timed out. The process will retry by itself.
  • Resolution: The process will retry by itself. 
• deadlock detected: there was an interaction with other active transactions that prevented the transaction from being completed.
  • Resolution: The process will retry by itself. 
• Invalid exponent, Value 'e', Pos 5, Type: Decimal: a SQL query is failing when executing your data processes, likely a syntax error in the definition of a Computation
  • Resolution: Please review the type and definition of the computations deployed. For instance, if your new computation type is number, make sure the fields it's based on are also exclusively numbers.
• Could not open relation with OID: query processing your data timed out. The process will retry by itself.
  • Resolution: The process will retry by itself. 

Connector status

The section Connectors - Pull indicates when was the last time MadKudu pulled data from your integration. MadKudu usually pulls every 5-10 min. 

Integration error codes

• No such column 'field_name' on entity 'Object': MadKudu is unable to pull any data for this object because this field was either deleted or its permission updated.
  • Resolution: Please remove the field from the list of fields pulled on MadKudu platform and update any mapping or model configuration using this field on the platform

• Invalid credentials MadKudu is unable to pull data from your integration because of invalid credentials
  • Resolution: Please go to app.madkudu.com > Integrations to update the credentials associated to the failing integration
• INVALID_LOGIN: Invalid username, password, security token; or user locked out: MadKudu is unable to pull data from your integration because of invalid credentials
  • Resolution: Please go to app.madkudu.com > Integrations to update the credentials associated to the failing integration
• Access denied: MadKudu does not have the necessary permissions to pull data from your integration.
  • Resolution: Please refer to the Integrations docs to see which permissions are necessary and update them in your integration.  
• This app hasn't been granted all required scopes to make this call: MadKudu does not have the necessary permissions to pull data from your integration.
  • Resolution: Please refer to the Integrations docs to see which permissions are necessary and update them in your integration.

• Too close to the API limit: MadKudu is unable to pull additional data because of the API limit of the integration.
  • Resolution: MadKudu will retry pulling data when there is fresh API quota. Consider increasing your API limits or the daily API bandwidth to prevent this error.
• Export daily quota 500MB exceeded (Marketo): MadKudu is unable to pull additional data because of the API limit of the integration. 
  • Resolution: MadKudu will retry pulling data when there is fresh API quota. Consider increasing your API limits or the daily API bandwidth to prevent this error.

• BAD_REFRESH_TOKEN, missing or invalid refresh token(HubSpot) the account associated with that refresh token uninstalled the MadKudu integration. 

  • Resolution: Please refer to the Integrations docs to re-install MadKudu in HubSpot
• InvalidBatch : Failed to process query: FUNCTIONALITY_NOT_ENABLED: Selecting compound data not supported in Bulk Query (Salesforce): MadKudu is unable to pull Compound fields (like Address, Geolocation, GeoCode ....) because of their limitations. 
  • Resolution: Please remove the field from the list of fields pulled on MadKudu platform.

For other errors, please refer to the error message you receive for more information.